Repository navigation
Surface workflow tasks that keep losing their lease (#1231) - #1244
Merged
Merged
Conversation
A pcntl queue:work kills a job at --timeout (60 s by default). A workflow task
killed there rolled back, stayed leased until its 300 s lease expired, and
task repair returned it to ready for the next worker to repeat. A task that
always needs longer than the timeout retried forever: attempt_count grew, no
failure was recorded and nothing told the operator why.
- Task repair counts each expired workflow-task lease on the task
(lost_lease_count, last_lease_lost_at; additive migration, written only
where the columns exist). The next claim logs a typed
workflow_task_lease_lost diagnostic. No history is written and the run
behaves as before.
- At workflows.v2.workflow_task_lost_leases.attention_threshold (3) lost
leases, RepairBlockedReason reports workflow_task_lease_lost ("Task Timing
Out", badge visible), so the run summary, describe and Waterline show repair
attention. Operator metrics add runs.workflow_task_lease_lost,
tasks.lease_lost_repeatedly and tasks.max_lost_lease_count.
- Opt-in workflow_task_lost_leases.fail_run (off by default): at the
threshold the next claim fails the run with WorkflowTaskLeaseLostException
(task_failure) before workflow code runs.
- A queue worker starting with pcntl and --timeout below the workflow task
lease logs one warning (WorkerStarting). The README documents running
workflow workers with --timeout=0 or above the longest workflow task.
V2WorkflowTaskLostLeaseTest covers counting, attention, describe, metrics,
the claim diagnostic, unchanged history, the opt-in failure and its
threshold, and the worker warning. Replay-regression evidence
workflow-task-lease-lost-fails-run-opt-in records the opt-in failure; its
consumer reproduces that history only with this change.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The public boundary scan forbids cross-tracker issue references in docs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…se-attention # Conflicts: # CHANGELOG.md
The state-machine journey upgrades from 2.33.0, which already has the compensation ledger, so a candidate that adds a migration runs it in a new batch. A plain migrate:rollback then undid only that newer batch and never reached the ledger's refusal, failing the journey for any added migration. expect_rollback_refused now rolls back the batch that holds the ledger (migrate:rollback --batch), which the rollback preflight already checks, so the journeys keep proving that retained scopes refuse a schema rollback. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This was referenced Nov 19, 2026
…imeout Review of the lost-lease change: - M1, migration lock: the columns are added outside the migrator transaction, each under the schema lock budget, and repeatably (2026_11_18_000100_add_lost_lease_tracking, as for the #863 column), so an ALTER behind an open task transaction fails fast instead of queueing reads. - M2, describe contract: the five repair_blocked_reason codes stay frozen, and can_repair and repair_attention are unchanged. Lost leases are reported in a separate additive task_attention field ({reason, consecutive, threshold, workflow_task_id, last_lease_lost_at}) on the run summary, list items and describe. Operator metrics add runs.task_attention (alongside tasks.lease_lost_repeatedly and tasks.max_lost_lease_count). With fail_run off, every field an existing consumer reads is identical at the threshold. - M3, timeouts: no blanket --timeout=0. RunWorkflowTask carries its own job timeout (WorkflowTaskJobTimeout): the lease less 30 s, or the configured workflow_task_job_timeout_seconds if lower, always 10 s below the connection's retry_after. A short worker --timeout no longer kills workflow tasks, and activities keep the worker's bounded --timeout. The worker-start warning now fires only for workers on the package's workflow queue, and only when retry_after caps workflow tasks. The README explains the lease, retry_after and timeout relationship and recommends a dedicated workflow-task connection. - Lows: neutral wording (the worker was killed, timed out, or lost its connection); fail_run refuses a threshold below 2 at boot; the schema check is re-done every 60 s and on any missing-column error (in its own savepoint, so it never aborts the caller's transaction), and degrades to untracked after a schema rollback, including SQLite's quirk of reading an unknown quoted column as a string. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The bounded claim, repair and fresh-run projections already hold the run's open workflow task, so they derive task_attention from it instead of querying the task table again. A bridge test pins that budget. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Workflow task jobs carry their own queue timeout only when workflows.v2.workflow_task_job_timeout_seconds is configured; unset, the worker's --timeout applies as before. The worker-start warning now fires when the timeout workflow tasks run under is below the lease, and says such tasks are retried after the lease expires and counted as lost leases. Summary writes tolerate a task_attention column dropped after the cached schema check. The README example routes activities to the bounded worker. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Implements the stuck-run proposal approved on #1231. It is separate from #1235. This revision applies the third review round: the workflow task job timeout is opt-in, the worker warning is reworded, summary writes tolerate a schema rollback, and the docs example routes activities to the bounded worker.
Problem
A workflow task whose worker is killed, times out or loses its connection loses its lease, and its work rolls back. It stays
leaseduntil the 300 s lease expires, and then task repair returns it to ready. With a short worker--timeout(60 s by default with pcntl), a long workflow task was killed on every retry.attempt_countgrew, no failure was recorded, and nothing was surfaced to the operator.Change
RunWorkflowTaskcarries no job timeout, so the worker's--timeoutapplies exactly as before. Only whenworkflows.v2.workflow_task_job_timeout_seconds(DW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS) is set does every workflow task job carry that timeout, kept 10 s below its connection'sretry_after; Laravel's worker then uses it in place of its--timeoutfor those jobs. There is no blanket--timeout=0. Workers of earlier releases that unserialize a new job ignore the extra property (SerializesModels::__unserializerestores only declared properties; checked undererror_reporting=-1with no deprecation), and the queue payload'stimeoutstays null when unset.--timeout, or the opt-in job timeout) is below the workflow task lease. It says such tasks are retried only after the lease expires and counted as lost leases, and how to raise--timeout(keepingretry_afterabove it) or setDW_V2_WORKFLOW_TASK_JOB_TIMEOUT_SECONDS.--timeout=0does not warn.lost_lease_count,last_lease_lost_at). The next claim logs a typedworkflow_task_lease_lostdiagnostic, worded neutrally: "worker was killed, timed out, or lost its connection". No history is written.workflows.v2.workflow_task_lost_leases.attention_threshold(3) consecutive lost leases, the run summary, list items and describe carry an additivetask_attentionfield:{reason: workflow_task_lease_lost, consecutive, threshold, workflow_task_id, last_lease_lost_at}. Operator metrics addruns.task_attention,tasks.lease_lost_repeatedlyandtasks.max_lost_lease_count. The fiverepair_blocked_reasoncodes stay frozen, andrepair_attentionandcan_repairare unchanged. The test asserts that every field an existing consumer reads is identical just below and at the threshold. Follow-ups to display or parse the field: Display the run summary's additive task_attention field waterline#339 and Parse the run summary's additive task_attention field server#1255.fail_runenabled, the claim at the threshold fails the run with a typedWorkflowTaskLeaseLostException(task_failure) before any workflow code runs. Boot refusesfail_runwith a threshold below 2.2026_11_18_000100_add_lost_lease_trackingsets$withinTransaction = false. Each column on the hotworkflow_tasksandworkflow_run_summariestables is added underwithSchemaLockBudget, as for Add opt-in deadline-triggered durable cleanup with an immutable total budget #863. Nothing is backfilled, and a repeated run skips completed steps.task_attentioncolumn dropped after the cached check forgets the cache and writes again without the field; the retry sits outside the projection's own transaction, and inside a caller's open PostgreSQL transaction the error is rethrown so the caller's retry sees the current schema.retry_afterrelate. Its example raises--timeoutfor the workers serving workflow tasks and routes activities to a bounded worker through the activity class's$queue(activities otherwise run on their workflow's queue), and it describes the opt-in from Workflow 2.34.0.docs/architecture/operational-liveness.mddescribestask_attentionbeside the frozen codes. The docs site is updated in Document the queue worker job timeout for workflow tasks durable-workflow.github.io#578.scripts/ci/upgrade-journey-lib.sh), so a PR that adds a migration no longer fails them.Tests and evidence
V2WorkflowTaskLostLeaseTest(weighted) covers:TaskWatchdogrepair pass, and the additive field at the threshold;timeoutis null), and when configured its value, theretry_aftercap, and the timeout carried by the dispatched job;--timeout=0;V2LostLeaseTrackingMigrationTest(weighted): no transaction, the lock budget used for each column, no backfill, and repeatable on populated tables.workflow-task-lease-lost-fails-run-opt-in: the cold runner replays it. The consumer reproduces that history only with this change.Not changed
With
fail_runoff, run semantics and every existing consumer field are unchanged. #1235 is untouched; the only overlap with it is.github/feature-test-timings.jsonandCHANGELOG.md.🤖 Generated with Claude Code